Handling partitioning skew in MapReduce using LEEN

MapReduce is emerging as a prominent tool for big data processing. Locality is a key feature in MapReduce that is extensively leveraged in dataintensive cloud system: it avoids network saturation when processing large amount of data by co-allocating computation and data storage — the map phase. However, our studies with Hadoop, a widely used MapReduce implementation, demonstrate that the presence of partitioning skew causes a huge amount of data transfer during the shuffle phase and leads to significant unfairness on the reduce input among different data nodes. As a result, the applications experience performance degradation due to the long data transfer during the shuffle phase along with the computation skew, particularly in reduce phase. In this paper, we develop a novel algorithm named LEEN for localityaware and fairness-aware key partitioning in MapReduce. LEEN embraces an asynchronous map and reduce scheme. All buffered intermediate keys are par⋆ corresponding author Shadi Ibrahim, Gabriel Antoniu INRIA Rennes-Bretagne Atlantique Rennes, France E-mail: {shadi.ibrahim, gabriel.antoniu}@inria.fr Hai Jin, Lu Lu, Song Wu Cluster and Grid Computing Lab Services Computing Technology and System Lab Huazhong University of Science and Technology Wuhan, China E-mail: [email protected] Bingsheng He School of Computer Engineering Nanyang Technological University Singapore E-mail: [email protected] 1 Partitioning skew refers to the case when a variation in either the intermediate keys’ frequencies or their distributions or both among different data nodes. 2 Shadi Ibrahim et al. titioned according to their frequencies and the fairness of the expected data distribution after the shuffle phase. We have integrated LEEN into Hadoop. Our experiments demonstrate that LEEN can efficiently achieve higher locality and reduce the amount of shuffled data. More importantly, LEEN guarantees fair distribution of the reduce inputs. As a result, LEEN achieves a performance improvement of up to 45% on different workloads.